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The manufacturing processes and the control systems for automatic 
production lines mainly are evaluated through usage of statistical methods as 
recently machine learning algorithms are also used. The aim of the paper is to 
present an approach for control measurement systems evaluation, based on a 
combination of statistical techniques like attribute repeatability & 
reproducibility analysis, measurement system analysis and supervised 
machine learning algorithms like random forest and KNN. The proposed 
method is verified in the production of the G8680x connector, which is used 
in the automotive industry. The control is performed 100% for all 
manufactured parts immediately after the “injection molding” process. It is 
proved that taking advantages of the statistics and machine learning, the 
manufacturing process and control measurement systems could be evaluated 
with very high accuracy. The exploration and analysis leads to the formulation 
of some recommendations in support of process engineers and managers. 
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1. INTRODUCTION 

An important element for evaluating the effectiveness and quality of control measurement systems, 
used in a manufacturing process, are the methods, which include an analysis of their capabilities. A well-known 
method for evaluating measuring systems is the repeatability & reproducibility (R&R) as information is 
obtained through graphical and tabular interpretation of statistical data [1], [2]. The application of the respective 
method depends on the type of control - quantitative or qualitative, manual or automatic, as well as on the 
conditions under which it is carried out (static or dynamic mode, also taking into account some environmental 
characteristics). The investigation related to the capabilities of measuring systems is associated with a 
comprehensive statistical study of the manufactured objects. The modern approach for ensuring quality 
measurements and control requires consideration of a wide variety of factors influencing the control 
measurement process. Various methods could be used to evaluate the quality of control measuring systems, 
which allow easy, fast and accurate assessment of the quality control and suitability of measuring instruments. 

Nowadays, the application of statistical and machine learning methods for increasing the quality and 
effectiveness of manufacturing process and control systems is an important topic for exploration, which results 
in influence on both manufacturer and customers. For better understanding the current research interests in the 
investigated area, the “big picture” related to the scientific production is built through bibliographic approach 
[3], [4] using VoSviewer software [5], [6] and the keywords: “repeatability & (and) reproducibility”, 
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learning” as it is presented on Figure 1. This investigation is done on 21 April 2021 as the bibliographic data 
is taken from the scientific database Scopus. The results from several queries outline the strong connections 
among the following terms: repeatability & reproducibility, measurement system analysis, statistical process 
control, metrology, quality control, design of experiments, control charts, additive manufacturing, ANOVA, 
structural equation modeling (SEM), others. 
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Figure 1. The constructed bibliometric map 


The created “main view” regarding the application of statistical and machine learning methods in 
manufacturing processes and control systems point out their important meaning in the era of big data for 
improving all parameters of control measurement systems and products manufacturing. This statement is also 
proved in detailed review of recently published scientific works. For example, Kuo and Huang [7] present the 
application of the R&R method for examination of the developed optical measurement system for measurement 
of thin film surface and their exploration proves its effectiveness. Baleani et al. discuss the importance of the 
R&R analysis at evaluation of the production line for cars assembly [8]. Barbosa et al. [9] use the R&R method 
to show the advantages of application of laser device technology in aircraft manufacturing process. Exploration 
about the importance of the measurement system capability for gathering the precise data is done in [10]. The 
authors explore the relationship among quality of measured data, quality control of products and quality 
analysis in manufacturing and usefulness of the R&R technique for their evaluation. Chang in his paper gives 
a new meaning of the method statistical process control (SPC) in the contemporary manufacturing environment 
which is characterized with big data collections [11]. Furthermore, Viharos and Jakab [12] connect SPC with 
reinforcement learning, which is a method typical for artificial intelligence and machine learning, with the aim 
to evaluate the performance of a manufacturing system. Khoza and Grobler [13] assess the quality control in a 
manufacturing process, comparing two techniques: SPC and machine learning. They conclude that the random 
forest algorithm is a better predictive method for quality control. Machine learning algorithms are used for 
analysis of datasets, gathered from a manufacturing process of semiconductors [14]. Better accuracy at solving 
a classification task is achieved through random forest and logistics regression. Another paper discusses the 
usage of a regression algorithm for purposes of a manufacturing process optimization [15]. It seems that 
machine learning is a suitable approach for analysis of a manufacturing process, which is characterized with 
performance variations of high-speed interconnections [16]. How a manufacturing process and its parameters 
to be improved through applying machine learning is also discussed in several other papers [17]-[19]. A review 
regarding machine learning and artificial intelligence application in manufacturing processes is performed in 
[20], where the authors draw the benefits and gaps. 
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It seems that statistics and machine learning are used for improving the manufacturing process, taking 
advantages of collected data sets and existing techniques and algorithms. The measuring and control phase of 
a product manufacturing could benefit from predictive and analytical models, created after applying suitable 
machine learning algorithm/s. It could point out some patterns, tendencies or anomalies that could be used or 
avoided for making better evaluation of a control measurement process. 

This work uses measurement system analysis (MSA), which by definition is an experimental and 
mathematical method used for variation identification in a measurement process [21], [22]. The purpose of the 
MSA is to statistically assess the capabilities of a control measurement system, which means to provide 
information on the reliability of the control. A quantitative indicator for this assessment is the total variation 
of the measurement process. The other important indicator is stability, i.e. maintaining reliability over time, 
including when control conditions are changing. Also, SPC is another applied method, which goal is to monitor 
and control a manufacturing process achieving its quality and effectiveness [23], [24]. 

The paper aims to present an approach for evaluation of an automatic measurement control system, 
which is based on advantages of statistical and machine learning methods. Its verification is done through the 
evaluation of the production of the G8680x connector, which is used in the automotive industry. The control 
is performed 100% for all manufactured parts immediately after the “injection molding” phase. The assembly 
and the types of control applied for determining the quality of the product, the way of their performance, as 
well as the criteria for suitability of the produced details are described. 


2. METHOD 

As it is seen, for evaluation of a control measurement system the data sets have to be prepared for 
statistical and machine learning analysis. For this purpose, firstly, the examined product line is described to 
outline the data sources and the data type. Secondly, based on the collected data, an analysis of the capabilities 
of the measurement control system is performed using the software product Develve [25]. Thirdly, the data 
sets are used for creation of predictive models through usage of machine learning algorithms in RapidMiner 
Studio [26]. 


2.1. Production line and product assembling 
The assembled products are G8680x connectors, which are part of a system for automatically tracking 
the distance to other cars while driving BMW cars (Figure 2). The production of those connectors is carried 
out in several steps as follows: 
a. Injection molding of a plastic body; 
b. Automatic assembly of metal terminals (CuSn6) in the plastic body; 
c. Automatic 100% control of the products for critical features for the functionality; 
d. Automatic package creation in roll packaging (belt with separate slots for each product). 


Figure 2. Connector G8680x 


The production line consists of 8 stations and can be conceptually divided into two important parts: 
the first one assembles the product and the second one performs the control. The assemble and control functions 
of the 8 stations are presented on Figure 3. The control functions of the stations concern whether the solder 
brackets are cut and pressed according to the specification, whether all terminals and brackets for soldering 
exist, whether the position of the pins is on the correct side for assembly, whether the position of the terminals 
is on the interface side and also high voltage short circuit check (1000V) is performed. After all these control 
functions, the product is correctly manufactured to this moment and the last Station 8 is responsible for 
transporting the product from the control part to the packing station as a signal for correct product is OK 
(correct) and for incorrect product is NOK (incorrect). 
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Figure 3. Assembling and control of the connector G8680x 


2.2. Statistical attribute R&R analysis by qualitative features - check for short or long leads 

The R&R analysis is performed taking into account the following indicator - check for short or long 
leads. The result of the check is pass/fail. The check is conducted with 20 samples, some of which are fit and 
the others unfit according to the respective indicator being checked. The samples are played at the station five 
times and a result is reported-"Yes" when the sample is manufactured according to the specification and "No" 
when it is unusable. The data collected is a total of 100 records. According to the internal rules of the company, 
in which the study was conducted, attribute R&R analysis on qualitative grounds is considered successful only 
when there is a 100% match of the results. In case of differences in the obtained result, a study is undertaken 
to establish the causes and their elimination. The study is performed by constructing various control charts 
(e.g., a series-feature map, a diagram of interactions, and stratifications by certain features). After finding and 
eliminating the causes, the attribute R&R analysis is performed again. 

In the present case, the R&R analysis is considered successful, as all samples have given a result that 
corresponds to the pre-established condition of the sample, respectively fit/unfit. The measurement and control 
is conducted according to following checks: 

a. Curved pin from the side interface 
It is checked by measuring the size 0 + 0.3/-Omm. The size tolerances are determined according to the 
customer’s requirement described in the customer’s drawing. This defect will cause the connector to be 
incorrectly assembled during the next manufacturing operation. The required reference standard samples 
of this station are OK and NOK with different values close to the limit 0.3 mm. 

b. Curved pin on the PCB side 
It is checked by measuring a size of 0 + 0.6/-Omm. The size tolerances are determined according to the 
customer’s requirement described in the specification. This defect will cause the connector to be incorrectly 
assembled during the next manufacturing operation. The required reference standard samples of this station 
are OK and NOK with different values close to the limit 0.3 mm. 

c. Check for braces 
The result of this check is pass/fail. The experiment was performed with 20 samples, some of which are fit, 
and the others are unfit according to the respective indicator, which is checked. In the specific case we 
prepare samples with and without braces. Table 1 presents the results after performing checks for braces 
by 2 operators as each operator conducts 2 different measurement tries. 

d. 1000V high voltage test 
This R&R analysis is performed according to the indicator - test for high voltage 1000V. The result of the 
check is pass/fail. The inspection was performed with 20 samples, some of which are suitable, and the 
others are unsuitable according to this indicator. 


2.3. Quality level check through measurement system analysis 

The study was performed with 30 connectors taken every 1 hour from the total set of manufactured 
parts. For this purpose, these samples are measured to the appropriate dimensions with a precision measuring 
instrument. In this case, a digital microscope was used - model CNC Quick Scope QS-250 — Mitutoyo. For this 
experiment the following parts are taken into consideration: pin1_interface2, pin3_interface2, pin1_interfacel 
and pin7_interfacel as the measurement is conducted by 8 operators. Part of the overall measurement results 
is shown in Table 2. In production conditions, the control is performed on each first shift or at the start of 
production immediately after the production line. 
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Table 1. Part of the gathered results at check for braces by 2 operators 


p ; Operator 1 checks Operator 2 checks 
Number Mastercheck Check] Check? Check 1 Check 2 
1 pass pass pass pass pass 
2 fail fail fail pass 
3 pass pass fail pass pass 
20 pass pass pass pass pass 


Table 2. Part of measurement results for 30 connectors by one operator 


Nanmber 1.02(0+0.6) 1.02(0+0.6) 1.02(0+0.6) 1.02(0+0.6) 
pinl_interface2__pin3_interface2__ pinl_interfacel _ pin7_interface2 
1 0.171 0.098 0.205 0.235 
2 0.132 0.105 0.352 0.245 
3 0.132 0.089 0.312 0.211 
4 0.100 0.095 0.385 0.019 
5 0.115 0.125 0.357 0.205 
30 0.108 0.110 0.311 0.203 


m) 


2531 


2.4. Machine learning predictive modeling 

From the above literature review it is seen that machine and deep learning is an accepted and utilized 
approach for evaluation or optimization of different phases of a manufacturing process, including for improving 
control systems. The most utilized algorithm is random forest and this is the reason it to be applied in this work. 
It will lead to understanding whether it is suitable to solve these specific classification tasks. 

Random forest is ensamble supervised machine learning algorithm, which uses bagging (bootstrap 
aggregation) technique based on creation of uncorrelated tree structures [27], [28]. The advantages of bagging 
are possibilities for combination of many trees (weak learners) for achieving smaller classification error, 
variance and bias, i.e. obtaining one strong learner according to the majority vote of the weak learners. The 
data samples from the data set are chosen in a random way and the tree nodes are aplited according to the 
randomly defined subsets of features. 

K-Nearest Neighbor (KNN) is also an algorithm from supervised machine learning, but it is called 
lazy learning algorithm, because the generalized result is delayed according to the input query [29], [30]. Its 
simplicity and effectiveness are among its advantages. This algorithm first finds the nearest neighbors to a 
given record and then determines the class of this record according to the majority vote or distance weighted 
vote. The workability specifics of random forest and KNN algorithms are presented through flowcharts 
respectively on Figures 4(a) and (b). 
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Figure 4. Flowcharts of (a) algorithms random forest and (b) KNN from supervised machine learning 
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3. RESULTS AND DISCUSSION 
3.1. Results from attribute repeatability & reproducibility analysis 

The attribute R&R analysis is focused on counts (discrete values) rather than measurements 
(continuous values). It points out whether a given sample pass or fail the check. In the ideal case R&R should 
be 100%, but in real checks 100%<R&R<90%. From the collected data, the repeatability of operator 1 is 
calculated to be 90% and for operator 2-95%. The reproducibility is obtained after comparing the master check 
to checks of operator | and 2. The received value of reproducibility is 85%. According to the criteria of the 
automotive industry action group [31], the control system is acceptable if R&R>90%. If 80%<R&R<90%, the 
control system is acceptable taking into account some factors for improvement. If R&R<80%, the control 
system is unacceptable. The control charts with data from checks conducted by operator 1 and operator 2 are 
presented on Figures 5(a) and (b). They confirm the values for repeatability of operator 1 (90%) and operator 
2 (95%). The reproducibility of the control system is shown through control chart on Figure 5(c) and it is 85%. 
The performed attribute R&R analysis indicates that the control system is acceptable, but some activities have 
to be done for its improvement, for example operators training or changing the control method to reduce the 
variations at control checks. 


(a) (b) (c) 
Figure 5. Results from attribute R&R analysis (a) repeatability of operator 1, (b) repeatability of operator 2, 
and (c) reproducibility of the control system 


3.2. Results from repeatability & reproducibility analysis 

The measurement results were processed with the statistical software product Develve. Repeatability 
is related to the variation at measurements on the same part performed by the same operator. Reproducibility 
gives the variation received after measurement conducted on the same part by different operators. In this work, 
for evaluation of the measurement control system, the instructions of the automotive industry action group are 
considered. If R&R<10%, then the measurement system is acceptable. If R&R is between 10% and 30%, then 
the measurement system may be acceptable, but it can be improved. If R&R>30%, then the measurement 
system is unacceptable and must be improved. The histograms of the measured parts (pin1_interface2, 
pin3_interface2, pin1_interface! and pin7_interfacel) by the same operator (repeatability) are presented on 
Figures 6(a)-(d) and the results point out that the R&R>30%, which means that the measurement system has 
to be revised to obtain normal distribution on the histograms. There are several techiques for data 
transformation and for reaching normal distribution and respectively R&R<10%, including a box-cox 
transformation, which is applied here. It can be seen on Figures 6(e)-(h) the improved histograms for 
repeatability after data processing according to the box-cox technique. The histograms with the measured 
results by eight operators are shown on Figures 7(a)-(h) (reproducibility). The histograms of reproducibility 
are also improved with box-cox transformation. Because the achieved results are similar to repeatability the 
improved reproducibility histograms are not shown. 


3.3. Machine learning predictive models 

The performed evaluation of the control measurement system in the previous section gives us 
information that this system is not perfect and it should be improved. It is possible this to be done through 
repeating the measurement by operators and obtaining more precise data that will cost more resources and 
operators’ effort. Also, it is possible the measurement instruments to be changed with more accurate ones. 
Another approach is based on statistical techniques, which could approximate data, achieving gaussian 
distribution and better R&R results. The last approach is taken into consideration for improvement of the 
control measurement system as it is presented in the above section. In this section is introduced the involvement 
of machine learning algorithms, which in combination with statistical techniques, are capable to evaluate a 
given control measurement system in a very precise way. The collected data from checks and measurements 
are used for creation of two predictive models in support of control measurement system evaluation: the first 
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one predicts whether the control system is i) acceptable, ii) acceptable with possible improvements or it is, iii) 
unacceptable according to the conducted attribute R&R analysis and the second one predicts whether the 
measurement system is i) acceptable, ii) maybe acceptable or iii) unacceptable taking into account the 
conducted measurement system analysis. 

The aim of the first predictive model is to prognoze whether the evaluated control system is 
acceptable, acceptable with possible improvements or it is unacceptable considering the pass/fail check 
according to the collected data during the attribute R&R analysis. Here a classification task is performed as 
learners are used random forest and KNN algorithms in RapidMined Studio. Three groups are formed: group 
A (the control system is acceptable when R&R>90%), AI (the control system is acceptable with possibility for 
improvement when 80%<R&R<90%) and UA (the control system is unacceptable when R&R<89%). The 
dataset is divided to 70% for learning and 30% for testing. The comparison of classifiers is presented through 
Table 3. It is seen that random forest algorithm is characterized with better accuracy and smaller errors 
according to the obtained results for KNN algorithm. It means that random forest is more suitable for solving 
this classification task, helping the evaluation of the control system. 


a 


(b) (d) 


Tl : 
(e) (f) (h) 


Figure 6. Statistical dependencies for the measured parts, produced with the statistical software Develve before 
and after the transformation technique (a) pin 1 interface 1, STDEV: 0.069, max: 0.579, min: 0.047, (b) pin 3 
interface 2, STDEV: 0.084, max: 0.476, min: 0.029, (c) pin 1 interface 1, STDEV: 0.066, max: 0.543, min: 
0.137, (d) pin 7 interface 1, STDEV: 0.071, max: 0.727, min: 0.019, (e) pin 1 interface 1, STDEV: 0.021, max: 
0.174, min: 0.094, (f) pin 3 interface 2, STDEV: 0.027, max: 0.143, min: 0.038, (g) pin 1 interface 1, STDEV: 
0.021, max: 0.172, min: 0.149, and (h) pin 7 interface 1, STDEV: 0.021, max: 0.127, min: 0.029 


(ec) (® (g) ~ (hy 


Figure 7. Histograms for the operators, obtained through the statistical software Develve (a) operator 1, 
STDEV: 0.094, max: 0.426, min: 0.019, (b) operator 2, STDEV: 0.051, max: 0.315, min: 0.101, (c) operator 3, 
STDEV: 0.084, max: 0.398, min: 0.108, (d) operator 4, STDEV: 0.094, max: 0.398, min: 0.101, (e) operator 5, 
STDEV: 0.050, max: 0.298, min: 0.107, (f) operator 6, STDEV: 0.094, max: 0.426, min: 0.019, (g) operator 7, 

STDEV: 0.039, max: 0.350, min: 0.176, and (h) operator 8, STDEV: 0.122, max: 0.727, min: 0.029 
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Table 3. Classifiers’ comparison 


Algorithm/parameter Accuracy (%) Classification error (%) Absolute error Relative error (%) Squared error 
Random Forest 83.33 16.6 0.092 9.24 0.047 
KNN 66.67 33.33 0.489 48.94 0.270 


The second model predicts whether the measurement system is acceptable (R&R<10%), maybe 
acceptable (10%<R&R<30%) and unacceptable (R&R>30%). The model is created on data used for 
measurement system analysis as three classification groups are formed: group A (acceptable measurement 
system), group MA (maybe acceptable) and group UA (unacceptable). As learners are used algorithms from 
supervised machine learning random forest and KNN. The classiffiers’ performance is compared as the results 
are summarized in Table 4. It seems that the used learners are suitable for conducting the correct classification 
and classes’ prediction as the accuracy is 95% for random forest and 90% for KNN. The evaluation of 
classiffiers’ performance is conducted considering the following parameters: 

a. Accuracy: It describes the correct predicted classes of the pin interface in percentages. 

b. Classification error: It presents the incorrect predicted classes of the pin interface in percentages. 

c. Absolute error: It shows the averaged absolute deviation of the predicted classes of the pin interface 
according to the original classes. 

d. Relative error: It describes the averaged value of the absolute deviation between the predicted and original 
classes of the pin interface divided to the original class. 

e. Squared error: It is the averaged value of the squared error. 


Table 4. Classifiers’ comparison 


Algorithm/parameter Accuracy (%) Classification error (%) Absolute error Relative error (%) Squared error 
Random Forest 95 5 0.030 2.97 0.017 
KNN 90 10 0.143 14.30 0.066 


A summarized graphics, presenting the proposed method for evaluation of control measurement 
system, is shown on Figure 8. It includes statistical techniques like attribute R&R analysis and measurement 
system analysis for evaluation of the current state of the control measurement system. Machine learning 
algorithms are used for predicting the quality of the control measurement system based on the historical data, 
taken from R&R and MSA. It contributes to improvement of the evaluation process and to redusing the usage 
of more resources and operators’ effort. 
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Figure 8. Proposed method for evaluation of control measurement system 


4. CONCLUSION 

The quality of the control measurement systems is an important topic for examination not only by 
researchers, but also by manufactures, process engineers and production managers. This is the reason for the 
current investigation that resulted in the proposition of a method for improving the evaluation process of control 
measurement systems. The method consists of a combination of statistical techniques and supervised machine 
learning algorithms. It is verified in the case of evaluation of automatic production line for G8680x connectors, 
used in the automotive industry. The method is capable of outlining the current state of the control measurement 
system as well as to predict it, which is in support of the responsible person for quality control and 
measurement, who is involved in decision making process and has to react in short time. 
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It is proved that supervised machine learning algorithms successfully can predict the group of the 
evaluated control measurement system as the predictive models are characterized with high accuracy. The 
algorithm random forest is more suitable for dealing with this classification task in comparison to the KNN 
algorithm. The performed exploration, experiments and analysis lead to the following recommendations: 1) the 
operator of the specific line should make a daily periodic inspection of the complex for control of each 
individual defect, using pre-prepared "golden" samples by a metrology specialist; ii) these samples should be 
checked periodically, every 3 months for validity and damage; iii) samples should be stored under special 
conditions to protect them from damage. The date of the subsequent inspection and the type of defect it inspects 
must be marked on each sample. The operator should monitor the date for subsequent inspection to avoid the 
use of samples out of control; iv) it is recommended that these tests be repeated over a period of time - for 
example 6 months. Due to the wear of the assets parts of the automatic line, over time this can lead to gaps and 
defects in production; v) in order to reduce the impact of wear of elements of the automatic line to introduce a 
schedule for preventive maintenance in accordance with the specifics of the line. Excessive wear of the 
elements will lead to unwanted gaps, which in turn will lead to poor quality products arriving at the customer; 
vi) by preparing an FMEA/risk analysis/, predictive maintenance and improvement can be organized, which 
will significantly reduce the risk of non-compliant production; vii) due to the weak cross-section of the 
terminals used for the connector, there is a risk of distortion. To ensure their correct position, the calibers used 
by the line must be checked periodically. The inspection is performed by measuring the functional dimensions 
by a metrology specialist; vii) in order to monitor the stability of the process of manufacturing (injection 
molding) of plastic bodies, to check the overall dimensions periodically during production; ix) during the 
examination of station 6 of the control complex, the appearance of a defect on the plastic part was noticed - 
separation of plastic shavings; x) the occurrence of this defect may lead to impossibility or difficulty during 
assembly with the counterpart of a subsequent step at the end customer. The cause of this defect is analyzed; 
Xi) it was concluded that due to the presence of eight sockets in the tool for the production of the plastic part, 
there are differences in size between them; xii) all the details are in the customer's requirements, but there is 
still a conflict between the caliber and the tested part; xiii) it is recommended to place a compensating device 
of the caliber, which will absorb the difference between the sockets and prevent the occurrence of this defect; 
xiv) the compensating device is a group of mechanical components between which there is a "soft" connection. 
This soft connection compensates for incorrectly positioned parts relative to each other with a difference in the 
dimensions of the opposite part. 
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